The Final Synthesis

This article concludes with some reflections on the state of the discipline of evaluation, reflections which synthesize a number of considerations. In this respect it is like many evaluations which close with an overview-a synthesis of preceding sub-conclusions-but it is unlike them in that its content consists of a discussion of the methodology of the last synthesis step in an evaluation-of whether and how to make it. In many evaluations, the lack of any explicit justification of the synthesis procedure is the Achilles heel of the whole effort. The data-gathering is impeccable, the analysis of documentation painstaking, the statistics flawless, most of the design state of the art-but the last step is simply a reaction to the rest, a high-inference judgment which is rather obviously idiosyncratic. Even in the best current product evaluation, usually a simpler matter than program evaluation, one quite often sees a conclusion which is a flying leap from the premises, essentially a non-sequitur. There-and throughout personnel evaluation-that last step is where the prejudices are most likely to show up, in the absence of rules governing the step, because prejudices feed on judgments. Sometimes, although much less often than is usually claimed, there is no way to avoid relying on judgment at this point. But, whether we call the last step clinical inference, intuition, professional judgment, connoisseurship, or impressionism, the solid body of the clinical vs. statistical research makes it clear that we rely on it at considerable peril. -Mat research shows how a very simple rule, if it is empirically-based, can beat expert judgment-including the combined judgment of a panel of experts-in almost all cases.